Image processing device, image processing method, and program for generating a focused image

ABSTRACT

The present technology relates to an image processing device, an image processing method, and a program that enable refocusing accompanied by desired optical effects. A light collection processing unit performs a light collection process to generate a processing result image focused at a predetermined distance, using images of a plurality of viewpoints. The light collection process is performed with the images of the plurality of viewpoints having pixel values adjusted with adjustment coefficients for the respective viewpoints. The present technology can be applied in a case where a refocused image is obtained from images of a plurality of viewpoints, for example.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a National Stage Patent Application of PCTInternational Patent Application No. PCT/JP2017/038469 (filed on Oct.25, 2017) under 35 U.S.C. § 371, which claims priority to JapanesePatent Application No. 2016-217763 (filed on Nov. 8, 2016), which areall hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present technology relates to an image processing device, an imageprocessing method, and a program, and more particularly, to an imageprocessing device, an image processing method, and a program forenabling refocusing accompanied by desired optical effects, for example.

BACKGROUND ART

A light field technique has been suggested for reconstructing, fromimages of a plurality of viewpoints, a refocused image, that is, animage captured with an optical system whose focus is changed, or thelike, for example (see Non-Patent Document 1, for example).

For example, Non-Patent Document 1 discloses a refocusing method using acamera array formed with 100 cameras.

CITATION LIST Non-Patent Document

-   Non-Patent Document 1: Bennett Wilburn et al., “High Performance    imaging Using Large Camera Arrays”

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

As for refocusing, the need for realizing refocusing accompanied byoptical effects desired by users and the like is expected to increase inthe future.

The present technology has been made in view of such circumstances, andaims to enable refocusing accompanied by desired optical effects.

Solutions to Problems

An image processing device or a program according to the presenttechnology is

an image processing device including: an acquisition unit that acquiresimages of a plurality of viewpoints; and a light collection processingunit that performs a light collection process to generate a processingresult image focused at a predetermined distance, using the images ofthe plurality of viewpoints, in which the light collection processingunit performs the light collection process using the images of theplurality of viewpoints, the images having pixel values adjusted withadjustment coefficients for the respective viewpoints, or

a program for causing a computer to function as such an image processingdevice.

An image processing method according to the present technology is animage processing method including: acquiring images of a plurality ofviewpoints; and performing a light collection process to generate aprocessing result image focused at a predetermined distance, using theimages of the plurality of viewpoints, in which the light collectionprocess is performed using the images of the plurality of viewpoints,the images having pixel values adjusted with adjustment coefficients forthe respective viewpoints.

In the image processing device, the image processing method, and theprogram according to the present technology, images of a plurality ofviewpoints are acquired, and a light collection process is performed togenerate a processing result image focused at a predetermined distance,using the images of the plurality of viewpoints. This light collectionprocess is performed with the images of the plurality of viewpoints, thepixel values of the images having been adjusted with adjustmentcoefficients for the respective viewpoints.

Note that the image processing device may be an independent device, ormay be an internal block in a single device.

Meanwhile, the program to be provided may be transmitted via atransmission medium or may be recorded on a recording medium.

Effects of the Invention

According to the present technology, it is possible to performrefocusing accompanied by desired optical effects.

Note that effects of the present technology are not limited to theeffects described herein, and may include any of the effects describedin the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example configuration of anembodiment of an image processing system to which the present technologyis applied.

FIG. 2 is a rear view of an example configuration of an image capturingdevice 11.

FIG. 3 is a rear view of another example configuration of the imagecapturing device 11.

FIG. 4 is a block diagram showing an example configuration of an imageprocessing device 12.

FIG. 5 is a flowchart showing an example process to be performed by theimage processing system.

FIG. 6 is a diagram for explaining an example of generation of aninterpolation image at an interpolation unit 32.

FIG. 7 is a diagram for explaining an example of generation of adisparity map at a parallax information generation unit 31.

FIG. 8 is a diagram for explaining an outline of refocusing through alight collection process to be performed by a light collectionprocessing unit 34.

FIG. 9 is a diagram for explaining an example of disparity conversion.

FIG. 10 is a diagram for explaining an outline of refocusing.

FIG. 11 is a flowchart for explaining an example of a light collectionprocess to be performed by the light collection processing unit 34.

FIG. 12 is a flowchart for explaining an example of an adjustmentprocess to be performed by an adjustment unit 33.

FIG. 13 is a diagram showing a first example of lens apertureparameters.

FIG. 14 is a diagram showing a second example of lens apertureparameters.

FIG. 15 is a diagram showing a third example of lens apertureparameters.

FIG. 16 is a diagram showing a fourth example of lens apertureparameters.

FIG. 17 is a diagram showing an example of filter parameters.

FIG. 18 is a block diagram showing another example configuration of theimage processing device 12.

FIG. 19 is a flowchart for explaining an example of a light collectionprocess to be performed by a light collection processing unit 51.

FIG. 20 is a block diagram showing an example configuration of anembodiment of a computer to which the present technology is applied.

MODES FOR CARRYING OUT THE INVENTION

<Embodiment of an Image Processing System to which the PresentTechnology is Applied>

FIG. 1 is a block diagram showing an example configuration of anembodiment of an image processing system to which the present technologyis applied.

In FIG. 1, the image processing system includes an image capturingdevice 11, an image processing device 12, and a display device 13.

The image capturing device 11 captures images of an object from aplurality of viewpoints, and supplies, for example, (almost) pan-focusimages obtained as a result of the capturing from the plurality ofviewpoints, to the image processing device 12.

The image processing device 12 performs image processing such asrefocusing for generating (reconstructing) an image focused on a desiredobject by using the captured images of the plurality of viewpointssupplied from the image capturing device 11, and supplies a processingresult image obtained as a result of the image processing, to thedisplay device 13.

The display device 13 displays the processing result image supplied fromthe image processing device 12.

Note that, in FIG. 1, the image capturing device 11, the imageprocessing device 12, and the display device 13 constituting the imageprocessing system can be all installed in an independent apparatus suchas a digital (still/video) camera, or a portable terminal like asmartphone or the like, for example.

Alternatively, the image capturing device 11, the image processingdevice 12, and the display device 13 can be installed in apparatusesindependent of one another.

Furthermore, any two devices among the image capturing device 11, theimage processing device 12, and the display device 13 can be installedin an apparatus independent of the apparatus in which the remaining oneapparatus is installed.

For example, the image capturing device 11 and the display device 13 canbe installed in a portable terminal owned by a user, and the imageprocessing device 12 can be installed in a server in a cloud.

Alternatively, some of the blocks of the image processing device 12 canbe installed in a server in a cloud, and the remaining blocks of theimage processing device 12, the image capturing device 11, and thedisplay device 13 can be installed in a portable terminal.

<Example Configuration of the Image Capturing Device 11>

FIG. 2 is a rear view of an example configuration of the image capturingdevice 11 shown in FIG. 1.

The image capturing device 11 includes a plurality of camera units(hereinafter also referred to as cameras) 21 _(i) that captures imageshaving the values of RGB as pixel values, for example, and the pluralityof cameras 21 _(i) captures images from a plurality of viewpoints.

In FIG. 2, the image capturing device 11 includes seven cameras 21 ₁, 21₂, 21 ₃, 21 ₄, 21 ₅, 21 ₆, and 21 ₇ as the plurality of cameras, forexample, and these seven cameras 21 ₁ through 21 ₇ are arranged in atwo-dimensional plane.

Further, in FIG. 2, the seven cameras 21 ₁ through 21 ₇ are arrangedsuch that one of the seven cameras 21 ₄ through 21 ₇, such as the camera21 ₁, for example, is disposed at the center, and the other six cameras21 ₂ through 21 ₇ are disposed around the camera 21 ₁, to form a regularhexagon.

Therefore, in FIG. 2, the distance between one camera 21 _(i) (i=1, 2, .. . , or 7) out of the seven cameras 21 ₁ through 21 ₇ and a camera 21_(j) (j=1, 2, . . . , or 7) closest to the camera 21 _(i) (the distancebetween the optical axes) is the same distance of B.

The distance of B between the cameras 21 _(i) and 21 _(j) may be about20 mm, for example. In this case, the image capturing device 11 can bedesigned to have almost the same size as the size of a card such as anIC card.

Note that the number of the cameras 21 _(i) constituting the imagecapturing device 11 is not necessarily seven, and it is possible toadopt a number from two to six, or the number eight or greater.

Also, in the image capturing device 11, the plurality of cameras 21 _(i)may be disposed at any appropriate positions, other than being arrangedto form a regular polygon such as a regular hexagon as described above.

Hereinafter, of the cameras 21 ₁ through 21 ₇, the camera 21 ₁ disposedat the center will be also referred to as the reference camera 21 ₁, andthe cameras 21 ₂ through 21 ₇ disposed around the reference camera 21 ₁will be also referred to as the peripheral cameras 21 ₂ through 21 ₇.

FIG. 3 is a rear view of another example configuration of the imagecapturing device 11 shown in FIG. 1.

In FIG. 3, the image capturing device 11 includes nine cameras 21 ₁₁through 21 ₁₉, and the nine cameras 21 ₁₁ through 21 ₁₇ are arranged inthree rows and three columns. Each of the 3×3 cameras 21 _(i) (i=11, 12,. . . , and 19) is disposed at the distance of B from an adjacent camera21 _(j) (j=11, 12, . . . , or 19) above the camera 21 _(i), below thecamera 21 _(i), or to the left or right of the camera 21 _(i).

In the description below, the image capturing device 11 includes theseven cameras 21 ₁ through 21 ₇ as shown in FIG. 2, for example, unlessotherwise specified.

Meanwhile, the viewpoint of the reference camera 21 ₁ is also referredto as the reference viewpoint, and a captured image PL1 captured by thereference camera 21 ₁ is also referred to as the reference image PL1.Further, a captured image PL #i captured by a peripheral camera 21 _(i)is also referred to as the peripheral image PL #i.

Note that the image capturing device 11 includes a plurality of cameras21 _(i) as shown in FIGS. 2 and 3, but may be formed with a microlensarray (MLA) as disclosed by Ren Ng and seven others in “Light FieldPhotography with a Hand-Held Plenoptic Camera”, Stanford Tech ReportCTSR 2005 February, for example. Even in a case where the imagecapturing device 11 is formed with an MLA, it is possible to obtainimages substantially captured from a plurality of viewpoints.

Further, the method of capturing images from a plurality of viewpointsis not necessarily the above method by which the image capturing device11 includes a plurality of cameras 21 _(i) or the method by which theimage capturing device 11 is formed with an MIA.

<Example Configuration of the image Processing Device 12>

FIG. 4 is a block diagram showing an example configuration of the imageprocessing device 12 shown in FIG. 1.

In FIG. 4, the image processing device 12 includes a parallaxinformation generation unit 31, an interpolation unit 32, an adjustmentunit 33, a light collection processing unit 34, and a parameter settingunit 35.

The image processing device 12 is supplied with captured images PL1through PL7 captured from seven viewpoints by the cameras 21 ₁ through21 ₇, from the image capturing device 11.

In the image processing device 12, the captured images PL #i aresupplied to the parallax information generation unit 31 and theinterpolation unit 323.

The parallax information generation unit 31 obtains parallax informationusing the captured images PL #i supplied from the image capturing device11, and supplies the parallax information to the interpolation unit 32and the light collection processing unit 34.

Specifically, the parallax information generation unit 31 performs aprocess of obtaining the parallax information between each of thecaptured images PL #i supplied from the image capturing device 11 andthe other captured images PL #j, as image processing of the capturedimages PL #i of a plurality of viewpoints, for example. The parallaxinformation generation unit 31 then generates a map in which theparallax information is registered for (the position of) each pixel ofthe captured images, for example, and supplies the map to theinterpolation unit 32 and the light collection processing unit 34.

Here, any appropriate information that can be converted into parallax,such as a disparity representing parallax with the number of pixels ordistance in the depth direction corresponding to parallax, can beadopted as the parallax information. In this embodiment, disparities areadopted as the parallax information, for example, and in the parallaxinformation generation unit 31, a disparity map in which the disparityis registered is generated as a map in which the parallax information isregistered.

Using the captured images PL1 through PL7 of the seven viewpoints of thecameras 21 ₁ through 21 ₇ from the image capturing device 11, and thedisparity map from the parallax information generation unit 31, theinterpolation unit 32 performs interpolation to generate images to beobtained from viewpoints other than the seven viewpoints of the cameras21 ₁ through 21 ₇.

Here, through the later described light collection process performed bythe light collection processing unit 34, the image capturing device 11including the plurality of cameras 21 ₁ through 21 ₇ can be made tofunction as a virtual lens having the cameras 21 ₁ through 21 ₇ as asynthetic aperture. In the image capturing device 11 shown in FIG. 2,the synthetic aperture of the virtual lens has a substantially circularshape with a diameter of approximately 2B connecting the optical axes ofthe peripheral cameras 21 ₂ through 21 ₇.

For example, where the viewpoints are a plurality of points equallyspaced in a square having the diameter of 2B of the virtual lens (or asquare inscribed in the synthetic aperture of the virtual lens), or theviewpoints are 21 points in the horizontal direction and 21 points inthe vertical direction, for example, the interpolation unit 32 performsinterpolation to generate a plurality of 21×21−7 viewpoints that are the21×21 viewpoints minus the seven viewpoints of the cameras 21 ₁ through21 ₇.

The interpolation unit 32 then supplies the captured images PL1 throughPL7 of the seven viewpoints of the cameras 21 ₁ through 21 ₇ and theimages of the 21×21−7 viewpoints generated by the interpolation usingcaptured images, to the adjustment unit 33.

Here, in the interpolation unit 32, the images generated by theinterpolation using captured images is also referred to as interpolationimages.

Further, the images of the 21×21 viewpoints, which are the total of thecaptured images PL1 through PL7 of the seven viewpoints of the cameras21 ₁ through 21 ₇ and the interpolation images of the 21×21−7 viewpointssupplied from the interpolation unit 32 to the adjustment unit 33, arealso referred to as viewpoint images.

The interpolation in the interpolation unit 32 can be considered as aprocess of generating viewpoint images of a larger number of viewpoints(21×21 viewpoints in this case) from the captured images PL1 through PL7of the seven viewpoints of the cameras 21 ₁ through 21 ₇. The process ofgenerating the viewpoint images of the large number of viewpoints can beregarded as a process of reproducing light beams entering the virtuallens having the cameras 21 ₁ through 21 ₇ as synthetic apertures fromreal-space points in the real space.

The adjustment unit 33 is supplied not only with the viewpoint images ofthe plurality of viewpoints from the interpolation unit 32, but alsowith adjustment parameters from the parameter setting unit 35. Theadjustment parameters are adjustment coefficients for adjusting pixelvalues, and are set for the respective viewpoints, for example.

The adjustment unit 33 adjusts the pixel values of the pixels of theviewpoint images of the respective viewpoints supplied from theinterpolation unit 32, using the adjustment coefficients for therespective viewpoints as the adjustment parameters supplied from theparameter setting unit 35. The adjustment unit 33 then supplies theviewpoint images of the plurality of viewpoints with the adjusted pixelvalues to the light collection processing unit 34.

Using the viewpoint images of the plurality of viewpoints supplied fromthe adjustment unit 33, the light collection processing unit 34 performsa light collection process that is image processing equivalent toforming an image of the object by collecting light beams that havepassed through an optical system such as a lens from the object, onto animage sensor or a film in a real camera.

In the light collection process by the light collection processing unit34, refocusing is performed to generate (reconstruct) an image focusedon a desired object. The refocusing is performed using the disparity mapsupplied from the parallax information generation unit 31 and a lightcollection parameter supplied from the parameter setting unit 35.

The image obtained through the light collection process by the lightcollection processing unit 34 is output as a processing result image to(the display device 13).

The parameter setting unit 35 sets a pixel of the captured image PL #i(the reference image PL1, for example) located at a position designatedby the user operating an operation unit (not shown), a predeterminedapplication, or the like, as the focus target pixel for focusing (orshowing the object), and supplies the focus target pixel as (part of)the light collection parameter to the light collection processing unit34.

The parameter setting unit 35 further sets adjustment coefficients foradjusting pixel values for each of the plurality of viewpoints inaccordance with an operation by the user or an instruction from apredetermined application, and supplies the adjustment coefficients forthe respective viewpoints as the adjustment parameters to the adjustmentunit 33.

The adjustment parameters are parameters for controlling pixel valueadjustment at the adjustment unit 33, and include the adjustmentcoefficients for each of the viewpoints of the viewpoint images to beused in the light collection process at the light collection processingunit 34, or for each of the viewpoints of the viewpoint images obtainedby the interpolation unit 32.

The adjustment parameters may be lens aperture parameters for achievingoptical image effects that can be actually or theoretically achievedwith an optical system such as an optical lens and a diaphragm, filterparameters for achieving optical image effects that can be actually ortheoretically achieved with a lens filter, or the like, for example.

Note that the image processing device 12 may be configured as a server,or may be configured as a client. Further, the image processing device12 may be configured as a server-client system. In a case where theimage processing device 12 is configured as a server-client system, someof the blocks of the image processing device 12 can be configured as aserver, and the remaining blocks can be configured as a client.

<Process to Be Performed by the Image Processing System>

FIG. 5 is a flowchart showing an example process to be performed by theimage processing system shown in FIG. 1.

In step S11, the image capturing device 11 captures images PL1 throughPL7 of seven viewpoints as a plurality of viewpoints. The capturedimages PL #i are supplied to the parallax information generation unit 31and the interpolation unit 32 of the image processing device 12 (FIG.4).

The process then moves from step S11 on to step S12, and the imageprocessing device 12 acquires the captured image PL #i from the imagecapturing device 11. Further, in the image processing device 12, theparallax information generation unit 31 performs a parallax informationgeneration process, to obtain the parallax information using thecaptured image PL #i supplied from the image capturing device 11, andgenerate a disparity map in which the parallax information isregistered.

The parallax information generation unit 31 supplies the disparity mapobtained through the parallax information generation process to theinterpolation unit 32 and the light collection processing unit 31, andthe process moves from step S12 on to step S13. Note that, in thisexample, the image processing device 12 acquires the captured images PL#i from the image capturing device 11, but the image processing device12 not only can acquire the captured images PL #i directly from theimage capturing device 11 but also can acquire, from a cloud, capturedimages PL #i that have been captured by the image capturing device 11 orsome other image capturing device (not shown), for example, and beenstored beforehand into the cloud.

In step S13, the interpolation unit 32 performs an interpolation processof generating interpolation images of a plurality of viewpoints otherthan the seven viewpoints of the cameras 21 ₁ through 21 ₇, using thecaptured images PL1 through PL7 of the seven viewpoints of the cameras21 ₁ through 21 ₇ supplied from the image capturing device 11, and thedisparity map supplied from the parallax information generation unit 31.

The interpolation unit 32 further supplies the captured images PL1through P17 of the seven viewpoints of the cameras 21 ₁ through 21 ₇supplied from the image capturing device 11 and the interpolation imagesof the plurality of viewpoints obtained through the interpolationprocess, as viewpoint images of a plurality of viewpoints to theadjustment unit 33. The process then moves from step S13 on to step S14.

In step S14, the parameter setting unit 35 sets the light collectionparameter and the adjustment parameters.

In other words, the parameter setting unit 35 sets adjustmentcoefficients for the respective viewpoints of the viewpoint images, inaccordance with a user operation or the like.

The parameter setting unit 35 also sets a pixel of the reference imagePL1 located at a position designated by a user operation or the like, asthe focus target pixel for focusing.

Here, the parameter setting unit 35 causes the display device 13 to alsodisplay, for example, a message prompting designation of the object ontowhich the reference image PL1 is to be focused among the captured imagesPL1 through PL7 of the seven viewpoints supplied from the imagecapturing device 11, for example. The parameter setting unit 35 thenwaits for the user to designate a position (in the object shown) in thereference image PL1 displayed on the display device 13, and then setsthe pixel of the reference image PL1 located at the position designatedby the user as the focus target pixel.

The focus target pixel can be set not only in accordance with userdesignation as described above, but also in accordance with designationfrom an application or in accordance with designation based onpredetermined rules or the like, for example.

For example, a pixel showing an object moving at a predetermined speedor higher, or a pixel showing an object moving continuously for apredetermined time or longer can be set as the focus target pixel.

The parameter setting unit 35 supplies the adjustment coefficients forthe respective viewpoints of the viewpoint images as the adjustmentparameters to the adjustment unit 33, and supplies the focus targetpixel as the light collection parameter to the light collectionprocessing unit 34. The process then moves from step S14 on to step S15.

In step S15, the adjustment unit 33 performs an adjustment process toadjust the pixel values of the pixels of the images of the respectiveviewpoints supplied from the interpolation unit 32, using the adjustmentcoefficients for the respective viewpoints as the adjustment parameterssupplied from the parameter setting unit 35. The adjustment unit 33supplies the viewpoint images of the plurality of viewpoints subjectedto the pixel value adjustment, to the light collection processing unit34, and the process then moves from step S15 on to step S16.

In step S16, the light collection processing unit. 34 performs a lightcollection process equivalent to collecting light beams that have passedthrough the virtual lens having the cameras 21 ₁ through 21 ₇ as thesynthetic aperture from the object onto a virtual sensor (not shown),using the viewpoint images of the plurality of viewpoints subjected tothe pixel value adjustment from the adjustment unit 33, the disparitymap from the parallax information generation unit 31, and the focustarget pixel as the light collection parameter from the parametersetting unit 35.

The virtual sensor onto which the light beams having passed through thevirtual lens are collected is actually a memory (not shown), forexample. In the light collection process, the pixel values of theviewpoint images of a plurality of viewpoints are integrated (as thestored value) in the memory as the virtual sensor, the pixel valuesbeing regarded as the luminance of the light beams gathered onto thevirtual sensor. In this manner, the pixel values of the image obtainedas a result of collection of the light beams having passed through thevirtual lens are determined.

In the light collection process by the light collection processing unit34, a reference shift amount BV (described later), which is a pixelshift amount for performing pixel shifting on the pixels of theviewpoint images of the plurality of viewpoints, is set. The pixels ofthe viewpoint images of the plurality of viewpoints are subjected topixel shifting in accordance with the reference shift amount BV, and arethen integrated. Thus, refocusing, or processing result imagegeneration, is performed to determine the respective pixel values of aprocessing result image focused on an in-focus point at a predetermineddistance.

As described above, the light collection processing unit 34 performs alight collection process (integration of (the pixel values of) pixels)on the viewpoint images of the plurality of viewpoints subjected to thepixel value adjustment. Thus, refocusing accompanied by various kinds ofoptical effects can be performed with the adjustment coefficients forthe respective viewpoints as the adjustment parameters for adjusting thepixel values.

Here, an in-focus point is a real-space point in the real space wherefocusing is achieved, and, in the light collection process by the lightcollection processing unit 34, the in-focus plane as the group ofin-focus points is set with focus target pixels as light collectionparameters supplied from the parameter setting unit 35.

Note that, in the light collection process by the light collectionprocessing unit 34, a reference shift amount By is set for each of thepixels of the processing result image. As a result, other than an imagefocused on an in-focus point at a distance, an image formed on aplurality of in-focus points at a plurality of distances can be obtainedas the processing result image.

The light collection processing unit 34 supplies the processing resultimage obtained as a result of the light collection process to thedisplay device 13, and the process then moves from step S16 on to stepS17.

In step S17, the display device 13 displays the processing result imagesupplied from the light collection processing unit 34.

Note that, although the adjustment parameters and the light collectionparameter are set in step 214 in FIG. 5, the adjustment parameters canbe set at any appropriate timing until immediately before the adjustmentprocess in step S15, and the light collection parameter can be set atany appropriate timing during the period from immediately after thecapturing of the captured images PL1 through PL7 of the seven viewpointsin step S11 till immediately before the light collection process in stepS15.

Further, the image processing device 12 (FIG. 4) can be formed only withthe light collection processing unit 34.

For example, in a case where the light collection process at the lightcollection processing unit 34 is performed with images captured by theimage capturing device 11 but without any interpolation image, the imageprocessing device 12 can be configured without the interpolation unit32. However, in a case where the light collection process is performednot only with captured images but also with interpolation images,ringing can be prevented from appearing in an unfocused object in theprocessing result image.

Further, in a case where parallax information about captured images of aplurality of viewpoints captured by the image capturing device 11 can begenerated by an external device using a distance sensor or the like, andthe parallax information can be acquired from the external device, forexample, the image processing device 12 can be configured without theparallax information generation unit 31.

Furthermore, in a case where the adjustment parameters can be set at theadjustment unit 33 while the light collection parameters can be set atthe light collection processing unit 34 in accordance with predeterminedrules or the like, for example, the image processing device 12 can beconfigured without the parameter setting unit 35.

<Generation of Interpolation Images>

FIG. 6 is a diagram for explaining an example of generation of aninterpolation image at the interpolation unit 32 shown in FIG. 4.

In a case where an interpolation image of a certain viewpoint is to begenerated, the interpolation unit 32 sequentially selects a pixel of theinterpolation image as an interpolation target pixel for interpolation.The interpolation unit 32 further selects pixel value calculation imagesto be used in calculating the pixel value of the interpolation targetpixel. The pixel value calculation images may be all of the capturedimages PL1 through PL7 of the seven viewpoints, or the captured imagesPL #i of some viewpoints close to the viewpoint of the interpolationimage. Using the disparity map from the parallax information generationunit 31 and the viewpoint of the interpolation image, the interpolationunit 32 determines the pixel (the pixel showing the same spatial pointas the spatial point shown on the interpolation target pixel, if imagecapturing is performed from the viewpoint of the interpolation image)corresponding to the interpolation target pixel from each of thecaptured images PL #1 of a plurality of viewpoints selected as the pixelvalue calculation images.

The interpolation unit 32 then weights the pixel value of thecorresponding pixel, and determines the resultant weighted value to bethe pixel value of the interpolation target pixel.

The weight used for the weighting of the pixel value of thecorresponding pixel may be a value that is inversely proportional to thedistance between the viewpoint of the captured image PL7 as the pixelvalue calculation image having the corresponding pixel and the viewpointof the interpolation image having the interpolation target pixel.

Note that, in a case where intense light with directivity is reflectedon the captured images PL #i, it is preferable to select captured imagesPL #i of some viewpoints such as three or four viewpoints as the pixelvalue calculation images, rather than selecting all of the capturedimages PL1 through PL7 of the seven viewpoints as the pixel valuecalculation images. With captured images PL #i of some of theviewpoints, it is possible to obtain an interpolation image similar toan image that would be obtained if image capturing is actually performedfrom the viewpoint of the interpolation image.

<Generation of a Disparity Map>

FIG. 7 is a diagram for explaining an example of generation of adisparity map at the parallax information generation unit 31 shown inFIG. 4.

In other words, FIG. 7 shows an example of captured images PL1 throughPL7 captured by the cameras 211 through 21 ₇ of the image capturingdevice 11.

In FIG. 7, the captured images PL1 through PL7 show a predeterminedobject obj as the foreground in front of a predetermined background.Since the captured images PL1 through PL7 have different viewpoints fromone another, the positions (the positions in the captured images) of theobject obj shown in the respective captured images P12 through P17differ from the position of the object obj shown in the captured imagePL1 by the amounts equivalent to the viewpoint differences, for example.

Here, the viewpoint (position) of a camera 21 _(i), or the viewpoint ofa captured image PL #i captured by a camera 21 _(i), is represented byvp #i.

For example, in a case where a disparity map of the viewpoint vp1 of thecaptured image PL1 is to be generated, the parallax informationgeneration unit 31 sets the captured image PL1 as the attention imagePL1 to which attention is paid. The parallax information generation unit31 further sequentially selects each pixel of the attention image PL1 asthe attention pixel to which attention is paid, and detects thecorresponding pixel (corresponding point)) corresponding to theattention pixel from each of the other captured images PL2 through PL7.

The method of detecting the pixel corresponding to the attention pixelof the attention image PL1 from each of the captured images PL2 throughPL7 may be a method utilizing the principles of triangulation, such asstereo matching or multi-baseline stereo, for example.

Here, the vector representing the positional shift of the correspondingpixel of a captured image PL #i relative to the attention pixel of theattention image PL1 is set as a disparity vector v #i, 1.

The parallax information generation unit 31 obtains disparity vectorsv2, 1 through v7, 1 for the respective captured images PL2 through PL7.The parallax information generation unit 31 then performs a majoritydecision on the magnitudes of the disparity vectors v2, 1 through v7, 1,for example, and sets the magnitude of the disparity vectors v #i, 1,which are the majority, as the magnitude of the disparity (at theposition) of the attention pixel.

Here, in a case where the distance between the reference camera 21 ₁ forcapturing the attention image P11 and each of the peripheral cameras 21₂ through 21 ₇ for capturing the captured images PL2 through PL7 is thesame distance of B in the image capturing device 11 as described abovewith reference to FIG. 2, when the real-space point shown in theattention pixel of the attention image PL1 is also shown in the capturedimages P12 through PL7, vectors that differ in orientation but are equalin magnitude are obtained as the disparity vectors v2, 1 through v7, 1.

In other words, the disparity vectors v2, 1 through v7, 1 in this caseare vectors that are equal in magnitude and are in the directionsopposite to the directions of the viewpoints vp2 through vp7 of theother captured images P12 through PL7 relative to the viewpoint vp1 ofthe attention image PL1.

However, among the captured images PL2 through PL7, there may be animage with occlusion, or an image in which the real-space pointappearing in the attention pixel of the attention image PL1 is hiddenbehind the foreground.

From a captured image PL #i that does not show the real-space pointshown in the attention pixel of the attention image PL1 (this capturedimage PL #i will be hereinafter also referred to as the occlusionimage), it is difficult to correctly detect the pixel corresponding tothe attention pixel.

Therefore, regarding the occlusion image PL #i, a disparity vector v #i,1 having a different magnitude from the disparity vectors v #j, 1 of thecaptured images PL #j showing the real-space point shown in theattention pixel of the attention image PL1 is obtained.

Among the captured images PL2 through PL7, the number of images withocclusion with respect to the attention pixel is estimated to be smallerthan the number of images with no occlusion. In view of this, theparallax information generation unit 31 performs a majority decision onthe magnitudes of the disparity vectors v2, 1 through v7, 1, and setsthe magnitude of the disparity vectors v #i, 1, which are the majority,as the magnitude of the disparity of the attention pixel, as describedabove.

In FIG. 7, among the disparity vectors v2, 1 through v7, 1, the threedisparity vectors v2, 1, v3, 1, and v7, 1 are vectors of the samemagnitude. Meanwhile, there are no disparity vectors of the samemagnitude among the disparity vectors v4, 1, v5, 1, and v6, 1.

Therefore, the magnitude of the three disparity vectors v2, 1, v3, 1,and v7, 1 are obtained as the magnitude of the disparity of theattention pixel.

Note that the direction of the disparity between the attention pixel ofthe attention image PL1 and any captured image PL #i can be recognizedfrom the positional relationship (such as the direction from theviewpoint vp1 toward the viewpoint vp #i) between the viewpoint vp1 ofthe attention image PL1 (the position of the camera 21 ₁) and theviewpoint vp #i of the captured image PL #i (the position of the camera21 _(i)).

The parallax information generation unit 31 sequentially selects eachpixel of the attention image P11 as the attention pixel, and determinesthe magnitude of the disparity. The parallax information generation unit31 then generates, as disparity map, a map in which the magnitude of thedisparity of each pixel of the attention image PL1 is registered withrespect to the position (x-y coordinate) of the pixel. Accordingly, thedisparity map is a map (table) in which the positions of the pixels areassociated with the disparity magnitudes of the pixels.

The disparity maps of the viewpoints vp #i of the other captured imagesPL #i can also be generated like the disparity map of the viewpoint vp#1.

However, in the generation of the disparity maps of the viewpoints vp #iother than the viewpoint vp #1, the majority decisions are performed onthe disparity vectors, after the magnitudes of the disparity vectors areadjusted on the basis of the positional relationship between theviewpoint vp #j a captured image PL #i and the viewpoints vp #1 of thecaptured images PL #j other than the captured image PL #i (thepositional relationship between the cameras 21 _(i) and 21 _(j)) (thedistance between the viewpoint vp #i and the viewpoint vp #j).

In other words, in a case where the captured image PL5 is set as theattention image PL5, and disparity maps are generated with respect tothe image capturing device 11 shown in FIG. 2, for example, thedisparity vector obtained between the attention image PL5 and thecaptured image PL2 is twice greater than the disparity vector obtainedbetween the attention image PL5 and the captured image PL1.

This is because, while the baseline length that is the distance betweenthe optical axes of the camera 21 ₃ for capturing the attention imagePL5 and the camera 21 ₁ for capturing the captured image PL1 is thedistance of B, the baseline length between the camera 21 ₅ for capturingthe attention image PL5 and the camera 21 ₂ for capturing the capturedimage PL2 is the distance of 2B.

In view of this, the distance of B, which is the baseline length betweenthe reference camera 21 _(i) and the other cameras 21 _(i), for example,is referred to as the reference baseline length, which is the referencein determining a disparity. A majority decision on disparity vectors isperformed after the magnitudes of the disparity vectors are adjusted sothat the baseline lengths can be converted into the reference baselinelength of B.

In other words, since the baseline length of B between the camera 21 ₅for capturing the captured image PL5 and the reference camera 21 ₁ forcapturing the captured image PL1 is equal to the reference baselinelength of B, for example, the magnitude of the disparity vector to beobtained between the attention image PL5 and the captured image PL1 isadjusted to a magnitude that is one time greater.

Further, since the baseline length of 2B between the camera 21 ₅ forcapturing the attention image PL5 and the camera 21 ₂ for capturing thecaptured image PL2 is equal to twice the reference baseline length of B,for example, the magnitude of the disparity vector to be obtainedbetween the attention image PL5 and the captured image PL2 is adjustedto a magnification that is ½ greater (a value multiplied by the ratiobetween the reference baseline length of B and the baseline length of 2Bbetween the camera 21 ₅ and the camera 21 ₂).

Likewise, the magnitude of the disparity vector to be obtained betweenthe attention image PL5 and another captured image PL #i is adjusted toa magnitude multiplied by the ratio to the reference baseline length ofB.

A disparity vector majority decision is then performed with the use ofthe disparity vectors subjected to the magnitude adjustment.

Note that, in the parallax information generation unit 31, the disparityof (each of the pixels of) a captured image PL #i can be determined withthe precision of the pixels of the captured images captured by the imagecapturing device 11, for example. Alternatively, the disparity of acaptured image PL #i can be determined with a precision equal to orlower than that of pixels having a higher precision than the pixels ofthe captured image PL #i (for example, the precision of sub pixels suchas ¼ pixels).

In a case where a disparity is to be determined with the pixel precisionor lower, the disparity with the pixel precision or lower can be used asit is in a process using disparities, or the disparity with the pixelprecision or lower can be used after being rounded down, rounded up, orrounded off to the closest whole number.

Here, the magnitude of a disparity registered in the disparity map ishereinafter also referred to as a registered disparity. For example, ina case where a vector as a disparity in a two-dimensional coordinatesystem in which the axis extending in a rightward direction is thex-axis while the axis extending in a downward direction is the y-axis, aregistered disparity is equal to the x component of the disparitybetween each pixel of the reference image PL1 and the captured image PL5of the viewpoint to the left of the reference image PL1 (or the xcomponent of the vector representing the pixel shift from a pixel of thereference image PL1 to the corresponding pixel of the captured imagePL5, the corresponding pixel corresponding to the pixel of the referenceimage PL1).

<Refocusing Through a Light Collection Process>

FIG. 8 is a diagram for explaining an outline of refocusing through alight collection process to be performed by the light collectionprocessing unit 34 shown in FIG. 4.

Note that, for ease of explanation, the three images, which are thereference image PL1, the captured image PL2 of the viewpoint to theright of the reference image PL1, and the captured image PL5 of theviewpoint to the left of the reference image PL1, are used as theviewpoint images of a plurality of viewpoints for the light collectionprocess in FIG. 8.

In FIG. 8, two objects obj1 and obj2 are shown in the captured imagesPL1, PL2, and PL5. For example, the object obj1 is located on the nearside, and the object obj2 is on the far side.

For example, refocusing is performed to focus on (or put the focus on)the object obj1 at this stage, so that an image viewed from thereference viewpoint of the reference image PL1 is obtained as thepost-refocusing processing result image.

Here, DP1 represents the disparity of the viewpoint of the processingresult image with respect to the pixel showing the object obj1 of thecaptured image PL1, or the disparity (of the corresponding pixel of thereference image PL1) of the reference viewpoint in this case. Likewise,DP2 represents the disparity of the viewpoint of the processing resultimage with respect to the pixel showing the object obj1 of the capturedimage PL2, and DP5 represents the disparity of the viewpoint of theprocessing result image with respect to the pixel showing the objectobj1 of the captured image PL5.

Note that, since the viewpoint of the processing result image is equalto the reference viewpoint of the captured image PL1 in FIG. 8, thedisparity DP1 of the viewpoint of the processing result image withrespect to the pixel showing the object obj1 of the captured image PL1is (0, 0).

As for the captured images PL1, PL2, and PL5, pixel shift is performedon the captured images PL1, PL2, and PL5 in accordance with thedisparities DP1, DP2, and DP5, respectively, and the captured images PL1and PL2, and PL5 subjected to the pixel shift are integrated. In thismanner, the processing result image focused on the object obj1 can beobtained.

In other words, pixel shift is performed on the captured images PL1,PL2, and PL5 so as to cancel the disparities DP1, DP2, and DP5 (thepixel shift being in the opposite direction from the disparities DP1,DP2, and DP5). As a result, the positions of the pixels showing obj1match among the captured images PL1, PL2, and PL5 subjected to the pixelshift.

As the captured images PL1, PL2, and PL5 subjected to the pixel shiftare integrated in this manner, the processing result image focused onthe object obj1 can be obtained.

Rote that, among the captured images PL1, P12, and PL5 subjected to thepixel shift, the positions of the pixels showing the object obj2 locatedat a different position from the object obj1 in the depth direction arenot the same. Therefore, the object obj2 shown in the processing resultimage is blurry.

Furthermore, since the viewpoint of the processing result image is thereference viewpoint, and the disparity DP1 is (0, 0) as described above,there is no substantial need to perform pixel shift on the capturedimage PL1.

In the light collection process by the light collection processing unit34, the pixels of viewpoint images of a plurality of viewpoints aresubjected to pixel shift so as to cancel the disparity of the viewpoint(the reference viewpoint in this case) of the processing result imagewith respect to the focus target pixel showing the focus target, and arethen integrated, as described above, for example. Thus, an imagesubjected to refocusing for the focus target is obtained as theprocessing result image.

<Disparity Conversion>

FIG. 9 is a diagram for explaining an example of disparity conversion.

As described above with reference to FIG. 7, the registrationdisparities registered in a disparity map are equivalent to the xcomponents of the disparities of the pixels of the reference image PL1with respect to the respective pixels of the captured image PL5 of theviewpoint to the left of the reference image PL1.

In refocusing, it is necessary to perform pixel shift on each viewpointimage so as to cancel the disparity of the focus target pixel.

Attention is now drawn to a certain viewpoint as the attentionviewpoint. In this case, the disparity of the focus target pixel of theprocessing result image with respect to the viewpoint image of theattention viewpoint, or the disparity of the focus target pixel of thereference image PL1 of the reference viewpoint in this case, is requiredin pixel shift of the captured image of the attention viewpoint, forexample.

The disparity of the focus target pixel of the reference image PL1 withrespect to viewpoint image of the attention viewpoint can be determinedfrom the registered disparity of the focus target pixel of the referenceimage PL1 (the corresponding pixel of the reference image PLcorresponding to the focus target pixel of the processing result image),with the direction from the reference viewpoint (the viewpoint of theprocessing result image) toward the attention viewpoint being taken intoaccount.

Here, the direction from the reference viewpoint toward the attentionviewpoint is indicated by a counterclockwise angle, with the x-axisbeing 0 [radian].

For example, the camera 21 ₂ is located at a distance equivalent to thereference baseline length of B in the +x direction, and the directionfrom the reference viewpoint toward the viewpoint of the camera 21 ₂ is0 [radian]. In this case, (the vector as) the disparity DP2 of the focustarget pixel of the reference image PL1 with respect to the viewpointimage (the captured image PL2) at the viewpoint of the camera 21 ₂ canbe determined to be (−RD, 0)=(−(B/B)×RD×cos θ, −(B/B)×RD×sin θ) from theregistered disparity RD of the focusing target pixel, as 0 [radian] isthe direction of the viewpoint of the camera 21 ₂.

Meanwhile, the camera 21 ₃ is located at a distance equivalent to thereference baseline length of B in the π/3 direction, for example, andthe direction from the reference viewpoint toward the viewpoint of thecamera 212 is π/3 [radian]. In this case, the disparity DP3 of the focustarget pixel of the reference image PL1 with respect to the viewpointimage (the captured image PL3) of the viewpoint of the camera 21 ₃ canbe determined to be (−RD×cos(π/3), −RD×sin(π/3))=(−(B/B)×RD×cos (π/3),−(B/B)×RD×sin (π/3)) from the registered disparity RD of the focustarget pixel, as the direction of the viewpoint of the camera 21 ₃ isπ/3 [radian].

Here, an interpolation image obtained by the interpolation unit 32 canbe regarded as an image captured by a virtual camera located at theviewpoint vp of the interpolation image. The viewpoint vp of thisvirtual camera is assumed to be located at a distance L from thereference viewpoint in the direction of the angle θ [radian]. In thiscase, the disparity DP of the focus target pixel of the reference imagePL1 with respect to the viewpoint image of the viewpoint vp (the imagecaptured by the virtual camera) can be determined to be (−(L/B)×RD×cosθ, −(L/B)×RD×sin θ) from the registered disparity RD of the focustarget, pixel, as the direction of the viewpoint vp as the angle θ.

Determining the disparity of a pixel of the reference image PL1 withrespect to the viewpoint image of the attention viewpoint from aregistered disparity RD and the direction of the attention viewpoint asdescribed above, or converting a registered disparity RD into thedisparity of a pixel of the reference image PL1 (the processing resultimage) with respect to the viewpoint image of the attention viewpoint,is also called disparity conversion.

In refocusing, the disparity of the focus target pixel of the referenceimage PL1 with respect to the viewpoint image of each viewpoint isdetermined from the registered disparity RD of the focus target pixelthrough disparity conversion, and pixel shift is performed on theviewpoint images of the respective viewpoints so as to cancel thedisparity of the focus target pixel.

In refocusing, pixel shift is performed on a viewpoint image so as tocancel the disparity of the focus target pixel with respect to theviewpoint image, and the shift amount of this pixel shift is alsoreferred to as the focus shift amount.

Here, in the description below, the viewpoint of the ith viewpoint imageamong the viewpoint images of a plurality of viewpoints obtained by theinterpolation unit 32 is also written as the viewpoint vp #i. The focusshift amount of the viewpoint image of the viewpoint vp #i is alsowritten as the focus shift amount DP #i.

The focus shift amount DP #i of the viewpoint image of the viewpoint vp#i can be uniquely determined from the registered disparity RD of thefocus target pixel through disparity conversion taking into account thedirection from the reference viewpoint toward the viewpoint vp #i.

Here, in the disparity conversion, (the vector as) a disparity(−(L/B)×RD×cos θ, −(L/B)×RD×sin θ) is calculated from the registereddisparity RD, as described above.

Accordingly, the disparity conversion can be regarded as an operation tomultiply the registered disparity RD by −(L/B)×cos θ and −(L/B)×sin θ,as an operation to multiply the registered disparity RD×−1 by (L/B)×cosθ and (L/B)×sin θ, or the like, for example.

Here, the disparity conversion can be regarded as an operation tomultiply the registered disparity RD×−1 by (L/B)×cos θ and (L/B)×sin θ,for example.

In this case, the value to be subjected to the disparity conversion,which is the registered disparity RD×−1, is the reference value fordetermining the focus shift amount of the viewpoint image of eachviewpoint, and will be hereinafter also referred to as the referenceshift amount BV.

The focus shift amount is uniquely determined through disparityconversion of the reference shift amount BV. Accordingly, the pixelshift amount for performing pixel shift on the pixels of the viewpointimage of each viewpoint in refocusing is substantially set depending onthe setting of the reference shift amount BV.

Note that, in a case where the registered disparity RD×−1 is adopted asthe reference shift amount BV as described above, the reference shiftamount By at a time when the focus target pixel is focused, or theregistered disparity RD of the focus target pixel×−1, is equal to the xcomponent of the disparity of the focus target pixel with respect to thecaptured image PL2.

<Light Collection Process>

FIG. 10 is a diagram for explaining refocusing through a lightcollection process

Here, a plane formed with a group of in-focus points (in-focusreal-space points in the real space) is set as an in-focus plane.

In a light collection process, refocusing is performed by setting anin-focus plane that is a plane in which the distance in the depthdirection in the real space is constant (does not vary), for example,and generating a processing result image focused on an object located onthe in-focus plane (or in the vicinity of the in-focus plane), usingviewpoint images of a plurality of viewpoints.

In FIG. 10, one person is shown in the near side while another person isshown in the middle in each of the viewpoint images of the plurality ofviewpoints. Further, a plane that passes through the position of theperson in the middle and is at a constant distance in the depthdirection is set as the in-focus plane, and a processing result imagefocused on an object on the in-focus plane, or the person in the middle,for example, is obtained from the viewpoint images of the plurality ofviewpoints.

Note that the in-focus plane may be a plane or a curved plane whosedistance in the depth direction in the real space varies, for example.Alternatively, the in-focus plane may be formed with a plurality ofplanes or the like at different distances in the depth direction.

FIG. 11 is a flowchart for explaining an example of a light collectionprocess to be performed by the light collection processing unit 34.

In step S31, the light collection processing unit 34 acquires(information about) the focus target pixel serving as a light collectionparameter from the parameter setting unit 35, and the process then moveson to step S32.

Specifically, the reference image PL1 or the like among the capturedimages PL1 through PL7 captured by the cameras 21 ₁ through 21 ₇ isdisplayed on the display device 13, for example. When the userdesignates a position in the reference image PL1, the parameter settingunit 35 sets the pixel at the position designated by the user as thefocus target pixel, and supplies (information indicating) the focustarget pixel as a light collection parameter to the light collectionprocessing unit 34.

In step S31, the light collection processing unit 34 acquires the focustarget pixel supplied from the parameter setting unit 35 as describedabove.

In step S32, the light collection processing unit 34 acquires theregistered disparity RD of the focus target pixel registered in adisparity map supplied from the parallax information generation unit 31.The light collection processing unit 34 then sets the reference shiftamount By in accordance with the registered disparity RD of the focustarget pixel, or sets the registered disparity RD of the focus targetpixel×−1 as the reference shift amount BV, for example. The process thenmoves from step S32 on to step 333.

In step 333, the light collection processing unit. 34 sets a processingresult image that is an image corresponding to one of viewpoint imagesof a plurality of viewpoints that have been supplied from the adjustmentunit 33 and been subjected to pixel value adjustment, such as an imagecorresponding to the reference image, or an image that has the same sizeas the reference image and has 0 as the initial value of the pixel valueas viewed from the viewpoint of the reference image, for example. Thelight collection processing unit 34 further determines the attentionpixel that is one of the pixels that are of the processing result imageand have not been selected as the attention pixel. The process thenmoves from step S33 on to step S34.

In step S34, the light collection processing unit 34 determines theattention viewpoint vp #i to be one viewpoint vp #i that has not beendetermined to be the attention viewpoint (with respect to the attentionpixel) among the viewpoints of the viewpoint images supplied from theadjustment unit 33. The process then moves on to step S35.

In step S35, the light collection processing unit 34 determines thefocus shift amounts DP #i of the respective pixels of the viewpointimage of the attention viewpoint vp #i, from the reference shift amountBy. The focus shift amounts DP #i are necessary for focusing on thefocus target pixel (put the focus on the object shown in the focustarget pixel).

In other words, the light collection processing unit 34 performsdisparity conversion on the reference shift amount BV by taking intoaccount the direction from the reference viewpoint toward the attentionviewpoint vp #i, and acquires the values (vectors) obtained through thedisparity conversion as the focus shift amounts DP #i of the respectivepixels of the viewpoint image of the attention viewpoint vp #i.

After that, the process moves from step S35 on to step S36. The lightcollection processing unit 34 then performs pixel shift on therespective pixels of the viewpoint image of the attention viewpoint vp#i in accordance with the focus shift amount DP #i, and integrates thepixel value of the pixel at the position of the attention pixel in theviewpoint image subjected to the pixel shift, with the pixel value ofthe attention pixel.

In other words, the light collection processing unit 34 integrates thepixel value of the pixel at a distance equivalent to the vector (forexample, the focus shift amount DP #i×−1 in this case) corresponding tothe focus shift amount DP #i from the position of the attention pixelamong the pixels of the viewpoint image of the attention viewpoint vp#i, with the pixel value of the attention pixel.

The process then moves from step S36 on to step S37, and the lightcollection processing unit 34 determines whether or not all theviewpoints of the viewpoint images supplied from the adjustment unit 33have been set as the attention viewpoint.

If it is determined in step S37 that not all the viewpoints of theviewpoint images from the adjustment unit 33 have been set as theattention viewpoint, the process returns to step S34, and thereafter, aprocess similar to the above is repeated.

If it is determined in step S37 that all the viewpoints of the viewpointimages from the adjustment unit 33 have been set as the attentionviewpoint, on the other hand, the process moves on to step S38.

In step S38, the light collection processing unit 34 determines whetheror not all of the pixels of the processing result image have been set asthe attention pixel.

If it is determined in step S38 that not all of the pixels of theprocessing result image have been set as the attention pixel, theprocess returns to step S33, and the light collection processing unit 34newly determines the attention pixel that is one of the pixels that areof the processing result image and have not been determined to be theattention pixel. After that, a process similar to the above is repeated.

If it is determined in step S38 that all the pixels of the processingresult image have been set as the attention pixel, on the other hand,the light collection processing unit 34 outputs the processing resultimage, and ends the light collection process.

Note that, in the light collection process shown in FIG. 11, thereference shift amount BV is set in accordance with the registereddisparity RD of the focus target pixel, and varies neither with theattention pixel nor with the attention viewpoint vp #i. In view of this,the reference shift amount DV is set, regardless of the attention pixeland the attention viewpoint vp #i.

Meanwhile, the focus shift amount DP #i varies with the attentionviewpoint vp #i and the reference shift amount BV. In the lightcollection process shown in FIG. 11, however, the reference shift amountBV varies neither with the attention pixel nor with the attentionviewpoint, vp #i, as described above. Accordingly, the focus shiftamount DP #i varies with the attention viewpoint vp #i, but does notvary with the attention pixel. In other words, the focus shift amount DP#i has the same value for each pixel of the viewpoint image of oneviewpoint, irrespective of the attention pixel.

In FIG. 11, the process in step S35 for obtaining the focus shift amountDP #i forms a loop for repeatedly calculating the focus shift amount. DP#i for the same viewpoint vp #i, regarding different attention pixels(the loop from step S33 to step S38). However, as described above, thefocus shift amount DP #i has the same value for each pixel of aviewpoint image of one viewpoint, regardless of the attention pixel.

Therefore, in FIG. 11, the process in step S35 for obtaining the focusshift amount DP #i is performed only once for one viewpoint.

In the light collection process shown in FIG. 11, the plane having aconstant distance in the depth direction is set as the in-focus plane,as described above with reference to FIG. 10. Accordingly, the referenceshift amount BV of the viewpoint image necessary for focusing on thefocus target pixel has such a value as to cancel the disparity of thefocus target pixel showing a spatial point on the in-focus plane havingthe constant distance in the depth direction, or the disparity of thefocus target pixel whose disparity is the value corresponding to thedistance to the in-focus plane.

Therefore, the reference shift amount BV depends neither on a pixel(attention pixel pixel) of the processing result image nor on theviewpoint (attention viewpoint) of a viewpoint image in which the pixelvalues are integrated, and accordingly, does not need to be set for eachpixel of the processing result image or each viewpoint of the viewpointimages (even if the reference shift amount BV is set for each pixel ofthe processing result image or each viewpoint of the viewpoint images,the reference shift amount By is set at the same value, and accordingly,is not, actually set for each pixel of the processing result image oreach viewpoint of the viewpoint images).

Note that, in FIG. 11, pixel shift and integration of the pixels of theviewpoint images are performed for each pixel of the processing resultimage. In the light collection process, however, pixel shift andintegration of the pixels of the viewpoint images can be performed foreach subpixel obtained by finely dividing each pixel of the processingresult image, other than for each pixel of the processing result image.

Further, in the light collection process shown in FIG. 11, the attentionpixel loop (the loop from step S33 to step S38) is on the outer side,and the attentional viewpoint loop (the loop from step S34 to step S37)is on the inner side. However, the attention viewpoint loop can be theouter-side loop while the attention pixel loop is the inner-side loop.

<Adjustment Process>

FIG. 12 is a flowchart for explaining an example of an adjustmentprocess to be performed by the adjustment unit 33 shown in FIG. 4.

In step S51, the adjustment unit 33 acquires the adjustment coefficientsfor the respective viewpoints as the adjustment parameters supplied fromthe parameter setting unit 35, and the process moves on to step S52.

In step S52, the adjustment unit 33 determines the attention viewpointvp #i to be one viewpoint vp #i that has not been determined to be theattention viewpoint among the viewpoints of the viewpoint imagessupplied from the interpolation unit 32. The process then moves on tostep S53.

In step S53, the adjustment unit 33 acquires the adjustment coefficientfor the attention viewpoint vp #i from among the adjustment coefficientsfor the respective viewpoints as the adjustment parameters supplied fromthe parameter setting unit 35, and the process moves on to step S54.

In step S54, the adjustment unit 33 determines the attention pixel to beone pixel among the pixels that have not been determined to be theattention viewpoint among the pixels of the viewpoint image of theattention viewpoint vp #i supplied from the interpolation unit 32. Theprocess then moves on to step S55.

In step S55, the adjustment unit 33 adjusts the pixel value of theattention pixel in accordance with the adjustment coefficient for theattention viewpoint vp #i, or multiplies the pixel value of theattention pixel by the adjustment coefficient for the attentionviewpoint vp #i and determines the resultant multiplied value to be thepixel value of the adjusted attention pixel, for example. The processthen moves on to step S56.

In step S56, the adjustment unit 33 determines whether or not all thepixels of the viewpoint image of the attention viewpoint vp #i have beenset as the attention pixel.

If it is determined in step S56 that not all of the pixels of theviewpoint image of the attention viewpoint vp #i have been set as theattention pixel, the process returns to step S54, and the adjustmentunit 33 newly determines the attention pixel that is one of the pixelsthat are of the viewpoint image of the attention viewpoint vp #i andhave not been determined to be the attention pixel. After that, aprocess similar to the above is repeated.

If it is determined in step S56 that all the pixels of the viewpointimage of the attention viewpoint vp #i have been set as the attentionpixel, on the other hand, the process moves on to step S57.

After the process moves on to step S57, the adjustment unit 33determines whether or not all the viewpoints of the viewpoint imagesfrom the interpolation unit 32 have been set as the attention viewpoint.

If it is determined in step S57 that not all the viewpoints of theviewpoint images from the interpolation unit 32 have been set as theattention viewpoint, the process returns to step S52, and thereafter, aprocess similar to the above is repeated.

On the other hand, if it is determined in step S57 that all theviewpoints of the viewpoint images from the interpolation unit 32 havebeen set as the attention viewpoint, or if all the pixel values of theplurality of viewpoint images from the interpolation unit 32 have beenadjusted, the adjustment unit 33 supplies the viewpoint images of allthe viewpoints with the adjusted pixel values to the light collectionprocessing unit 34, and ends the adjustment process.

The light collection process shown in FIG. 11 (the integration of thepixel values of the pixels of the viewpoint images of the plurality ofviewpoints in step S36) is performed on the viewpoint images of theplurality of viewpoints with the adjusted pixel values, which areobtained through the above adjustment process.

Accordingly, the coefficients corresponding to the optical effects areadopted as the adjustment coefficient for the respective viewpoints asthe adjustment parameters, so that refocusing accompanied by variousoptical effects can be performed.

In the description below, the adjustment coefficients for the respectiveviewpoints as the adjustment parameters will be explained throughexamples of lens aperture parameters for achieving optical image effectsthat can be actually or theoretically achieved with an optical systemsuch as an optical lens and a diaphragm, and filter parameters forachieving optical image effects that can be actually or theoreticallyachieved with a lens filter.

<Lens Aperture Parameters>

FIG. 13 is a diagram showing a first example of lens apertureparameters.

Here, the total number of viewpoints of the viewpoint images obtained bythe interpolation unit 32 is assumed to be M², which is M viewpoints inthe horizontal direction and M viewpoints in the vertical direction.

The transmittances set for the respective viewpoints of the M×Mviewpoints can be adopted as the adjustment coefficients for therespective viewpoints of the M×M viewpoints as lens aperture parameters.

To set the transmittances for the respective viewpoints, thedistribution of the transmittance that produce desired lens anddiaphragm effects is divided into M×M blocks in the same number as theM×M viewpoints, for example, and the representative value of thetransmittance of each block is determined. The representative value (arepresentative value being the mean value, the median, or the like ofthe transmittances in a block, for example) of the block that is the xthblock from the left and the yth block from the bottom (this block isalso referred to as the (x, y)th block) is set as the transmittance ofthe (x, y)th viewpoint.

FIG. 13 shows the distribution of the transmittances that produce theeffects of a smooth transfer focus (STF) lens. More specifically, FIG.13 shows a plan view of the transmittances set for the respectiveviewpoints in accordance with the distribution of the transmittancesamong which the transmittance at the center is the highest and thetransmittances at the farthest peripheral portions are the lowest, and across-sectional view of the transmittances for the respectiveviewpoints, taken along a line segment LO.

Here, the planar shape (the shape appearing in the plan view) of thedistribution of the transmittances that produce the effects of an STFlens is almost circular, but the line segment LO passes through thecenter of the circle and extends parallel to the x direction (horizontaldirection).

Further, in the plan view shown in FIG. 13, the differences in tone(grayscale) indicate the transmittances. The darker the tone, the lowerthe transmittance.

These also apply to the plan views shown in FIGS. 14, 15, and 16, whichwill be described later.

With the adjustment coefficients as the transmittances that are set forthe respective viewpoints in accordance with the distribution of thetransmittances that produce the effects of an STF lens, refocusing canbe performed to realize natural blurring that varies softly in itsdegree of blurriness in the direction from the center toward theperiphery of the blurred portion, like blurring that can be realizedwith an STF lens.

FIG. 14 is a diagram showing a second example of lens apertureparameters.

FIG. 14 shows the distribution of the transmittances that produce theeffects of a mirror lens More specifically, FIG. 14 shows a plan view ofthe transmittances set for the respective viewpoints in accordance withthe distribution of the transmittances among which the transmittance ata portion slightly shifted toward the central portion from the peripheryis the highest, and the transmittance decreases in a direction towardthe central portion or toward the periphery, and a cross-sectional viewof the transmittances for the respective viewpoints, taken along theline segment LO.

With the adjustment coefficients as the transmittances that are set forthe respective viewpoints in accordance with the distribution of thetransmittances that produce the effects of a mirror lens, refocusing canbe performed to realize ring blurring or double-line blurring, likeblurring that can be realized with a mirror lens.

FIG. 15 is a diagram showing a third example of lens apertureparameters.

FIG. 15 shows a plan view of transmittances set for the respectiveviewpoints in accordance with a distribution that is generated bymodifying the distribution of transmittances that produce the effects ofan STF lens so as to reduce the size of the circle as the planar shapeof the distribution of the transmittances that produce the effects of anSTF lens shown in FIG. 13 (this modified distribution will behereinafter also referred to as the STF modified distribution). FIG. 15also shows a cross-sectional view of the transmittances for therespective viewpoints, taken along the line segment 10.

Note that, in FIG. 13, to produce the effects of a diaphragm in an openstate, the transmittance distribution is not particularly controlled. InFIG. 15, however, to produce the effects of a diaphragm in a narrowedstate, the transmittances for the viewpoints outside a circle slightlylarger than the circle as the planar shape of the STF modifieddistribution, or the transmittances for the viewpoints blocked fromlight beams by the diaphragm in the narrowed state, are set (controlled)to 0%.

With the above adjustment coefficients as the transmittances for therespective viewpoints, refocusing can be performed to realize naturalblurring that varies softly in its degree of blurriness in the directionfrom the center toward the periphery of the blurred portion, likeblurring that can be realized with an STF lens.

Further, an image with a deep depth of field can be obtained as thepost-refocusing processing result image.

In other words, an image having a deep depth of field and blurrinessrealized with an STF lens can be obtained as the post-refocusingprocessing result image.

Note that, even if image capturing is performed with an actual STF lenshaving its diaphragm narrowed, it would be difficult to obtain an imagehaving a deep depth of field, and natural blurriness to be realized withan STF lens.

In other words, in a captured image obtained in a case where imagecapturing is performed with an actual STF lens having its diaphragmnarrowed, the depth of field is deepened by the diaphragm in thenarrowed state.

However, in a case where image capturing is performed with an actual STFlens having its diaphragm narrowed, light beams that are passing throughthe STF lens region (a region with low transmittances) equivalent to aregion other than the central portion of the circle as the planar shapeof the distribution of transmittances that produce the effects of an STFlens shown in FIG. 13 are blocked by the diaphragm in the narrowedstate. Therefore, it is difficult to achieve blurriness similar to thenatural blurriness to be achieved with an STF lens having a diaphragmnot in a narrowed state.

FIG. 16 is a diagram showing a fourth example of the lens apertureparameter.

FIG. 16 shows a plan view of transmittances set for the respectiveviewpoints in accordance with a distribution that is generated bymodifying the distribution of transmittances that produce the effects ofa mirror lens so as to reduce the size of the planar shape of thedistribution of the transmittances that produce the effects of a mirrorlens shown in FIG. 14 (this modified distribution will be hereinafteralso referred to as the mirror-lens modified distribution). FIG. 16 alsoshows a cross-sectional views of the transmittances for the respectiveviewpoints, taken along the line segment LO.

Note that, in FIG. 14, to produce the effects of a diaphragm in an openstate, the transmittance distribution is not particularly controlled. InFIG. 16, however, to produce the effects of a diaphragm in a narrowedstate, the transmittances for the viewpoints outside a circle slightlylarger than the circle as the planar shape of the STF modifieddistribution, or the transmittances for the viewpoints blocked fromlight beams by the diaphragm in the narrowed state, are set (controlled)to 0%, as in FIG. 15.

With the above adjustment coefficients as the transmittances for therespective viewpoints, refocusing can be performed to realize ringblurring or double-line blurring, like blurring that can be realizedwith a mirror lens.

Further, an image with a deep depth of field can be obtained as thepost-refocusing processing result image.

In other words, an image having a deep depth of field and blurrinessrealized with a mirror lens can be obtained as the post-refocusingprocessing result image.

Note that, even if image capturing is performed with an actual mirrorlens having its diaphragm narrowed, it would be difficult to obtain animage having a deep depth of field, and ring blurring or double-lineblurring to be realized with a mirror lens.

In other words, in a captured image obtained in a case where imagecapturing is performed with an actual mirror lens having its diaphragmnarrowed, the depth of field is deepened by the diaphragm in thenarrowed state.

However, in a case where image capturing is performed with an actualmirror lens having its diaphragm narrowed, light beams that are passingthrough the mirror lens region (the region with the highesttransmittance and its vicinity region) equivalent to a region other thanthe central portion of the circle as the planar shape of thedistribution of transmittances that produce the effects of a mirror lensshown in FIG. 14 are blocked by the diaphragm in the narrowed state.Therefore, it is difficult to achieve blurriness similar to the ringblurriness or double-line blurriness to be achieved with a mirror lenshaving a diaphragm not in a narrowed state.

In a case where the above lens aperture parameters are adopted as theadjustment coefficients for the respective viewpoints, the adjustmentcoefficients as the lens aperture parameters are denoted by a, and thepixel values of the pixels of the viewpoint images obtained by theinterpolation unit 32 are denoted by I. In such a case, the adjustmentunit 33 performs an adjustment process for adjusting the pixel values I,by determining pixel values α×I to be the pixel values after theadjustment of the pixel values I, for example.

The light collection processing unit 34 then performs a light collectionprocess on the viewpoint images subjected to the above adjustmentprocess, to perform refocusing reflecting desired lens blurriness and adesired narrowed aperture state.

Note that, in FIGS. 13 through 16, the adjustment coefficients for therespective viewpoints are set in accordance with the distribution oftransmittances that produce the effects of an STF lens or a mirror lens.However, the adjustment coefficients for the respective viewpoints maybe set in accordance with the distribution of transmittances thatproduce the effects of some other lens.

Further, in FIGS. 15 and 16, the distribution of transmittances iscontrolled so as to produce the effects of an aperture in a narrowedstate. However, the distribution of transmittances may be adjusted so asto produce the effects of an aperture in any desired state.

Still further, in FIGS. 13 through 16, a transmittance distributionhaving a substantially circular planar shape is used in setting theadjustment coefficients for the respective viewpoints. However, atransmittance distribution modified into a desired shape such as a heartshape or a stellar shape as its planar shape, for example, can be usedin setting the adjustment coefficients for the respective viewpoints. Inthis case, a processing result image in which a desired shape appears inthe blurring can be obtained.

<Filter Parameters>

FIG. 17 is a diagram showing an example of filter parameters.

In a case where image capturing is performed with an actual single lenscamera or the like, a gradation filter such as a color effect filter ora peripheral effect filter having gradations may be used as a lensfilter provided in front of a lens.

FIG. 17 shows an example of a gradation filter, and an example ofadjustment coefficients for the respective viewpoints as filterparameters that are set in accordance with the distribution of gainsthat produce the filter effects of the gradation filter.

In FIG. 17, the total number of viewpoints of the viewpoint imagesobtained by the interpolation unit 32 is 5², which is M×M=5×5viewpoints.

Gains that are set for the respective viewpoints of the M×N viewpointsand are for luminance or a predetermined color may be adopted as theadjustment coefficients for the respective viewpoints of the M×Mviewpoints as the filter parameters.

The gains for the respective viewpoints can be set by dividing thedistribution of the gains that produce desired filtering effects intoN×M blocks in the same number as M×M, determining the representativevalue of the gain of each block, and setting the representative value ofthe (x, y)th block as the gain for the (x, y)th viewpoint, for example.

In FIG. 17, the gains as the adjustment coefficients for the respectiveviewpoints of the M×M=5×5 viewpoints are set in accordance with thedistribution of the gains that produce the filter effects of a bluegradation filter.

Here, in the gradation filter shown in FIG. 17, tone indicates the gainwith respect to the blue color. The darker the tone, the higher thegain.

The gradation filter shown in FIG. 17 is a filter that has a higher gainwith respect to the blue color on the upper side.

In a case where the above film parameters are adopted as the adjustmentcoefficients for the respective viewpoints, the adjustment coefficientsas the filter parameters are denoted by G, and the red, green, and blue(RGB) components as the pixel values of the pixels of the viewpointimages obtained by the interpolation unit 32 are denoted by (Ir, Ig,Ib). In such a case, the adjustment unit 33 performs an adjustmentprocess for adjusting the pixel values (Ir, Ig, Ib), by determiningpixel values (Ir, Ig, Ib×G) to be the pixel values after the adjustmentof the pixel values (Ir, Ig, Ib), for example.

With the above adjustment coefficients as the gains for the respectiveviewpoints set in accordance with the distribution of the gains thatproduce the filter effects of a gradation filter, refocusing can beperformed to obtain a processing result image in which the degree ofblueness is higher at an upper portion.

Note that, in FIG. 17, the adjustment coefficients for the respectiveviewpoints as the filter parameters are set in accordance with thedistribution of the gains of a gradation filter having higher gains withrespect to the blue color at upper portions. However, the adjustmentcoefficients for the respective viewpoints may be set in accordance withthe distribution of the gains that produce filter effects other than thefilter effects of the gradation filter shown in FIG. 17. The gains maybe with respect to luminance or a desired color (which is not the bluecolor, but is the red color, the green color, or the like, for example).

<A Other Example Configuration of the Image Processing Device 12>

FIG. 18 is a block diagram showing another example configuration of theimage processing device 12 shown in FIG. 1.

Note that, in the drawing, the components equivalent to those in FIG. 4are denoted by the same reference numerals as those used in FIG. 4, andexplanation thereof is not repeated herein.

The image processing device 12 in FIG. 18 includes the parallaxinformation generation unit 31, the interpolation unit 32, the parametersetting unit 35, and a light collection processing unit 51.

Accordingly, the image processing device 12 in FIG. 18 is the same asthat in the case shown in FIG. 4 in including the parallax informationgeneration unit 31, the interpolation unit 32, and the parameter settingunit 35.

However, the image processing device 12 in FIG. 18 differs from that inthe case shown in FIG. 4 in that the adjustment unit 33 is not provided,and the light collection processing unit 51 is provided in place of thelight collection processing unit 34.

In FIG. 4, the pixel values of the pixels of the viewpoint images areadjusted by the adjustment unit 33, and the light collection process isperformed on the viewpoint images after the adjustment of the pixelvalues. In the image processing device 12 in FIG. 18, on the other hand,the pixel values to be subjected to integration are adjusted immediatelybefore the integration of the pixel values of the pixels of theviewpoint images is performed, and the pixel value integration isperformed on the adjusted pixel values in the light collection process.

In FIG. 18, the light collection processing unit 51 performs a lightcollection process similar to that performed by the light collectionprocessing unit 34 shown in FIG. 4, but further adjusts the pixel valuesof the pixels of the viewpoint images in the light collection process.Therefore, in addition to the light collection parameters to be used inthe light collection process, the adjustment parameters to be used foradjusting the pixel values are supplied from the parameter setting unit35 to the light collection processing unit 51.

In the light collection process, the light collection processing unit 51adjusts the pixel values of the pixels of the viewpoint imagesimmediately before integrating the pixel values of the pixels of theviewpoint images, and performs the pixel value integration on theadjusted pixel values.

FIG. 19 is a flowchart for explaining an example of a light collectionprocess to be performed by the light collection processing unit 51.

In step S71, the light collection processing unit 51 acquires a focustarget pixel as a light collection parameter from the parameter settingunit 35, as in step S31 in FIG. 11.

In step S71, the light collection processing unit 51 further acquiresthe adjustment coefficients for the respective viewpoints as theadjustment parameters from the parameter setting unit 35, and theprocess then moves on to step S72.

In steps S72 through S75, the light collection processing unit 51performs processes similar to the respective processes in steps S32through S35 in FIG. 11, to determine the focus shift amount DP #i of theattention viewpoint vp #i.

The process then moves from step S75 on to step S76, and the lightcollection processing unit 51 acquires the adjustment coefficient, forthe attention viewpoint vp #i from among the adjustment coefficients forthe respective viewpoints as the adjustment parameters from theparameter setting unit 35. The process then moves on to step S77.

In step S77, the light collection processing unit 51 sets the adjustmenttarget pixel that is the pixel at a distance equivalent to the vector(for example, the focus shift amount DP #i×−1 in this case)corresponding to the focus shift amount DP #i from the position of theattention pixel among the pixels of the viewpoint image of the attentionviewpoint vp #i. The light collection processing unit 51 then adjuststhe pixel value of the adjustment target pixel in accordance with theadjustment coefficient for the attention viewpoint vp #i, or multipliesthe pixel value of the adjustment target pixel by the adjustmentcoefficient for the attention viewpoint vp #i and determines theresultant multiplied value to be the adjusted pixel value of theadjustment target pixel, for example. The process then moves from stepS77 on to step S78.

In step S78, the light collection processing unit 51 performs pixelshift on the respective pixels of the viewpoint image of the attentionviewpoint vp #i in accordance with the focus shift amount DP #i, andintegrates the pixel value of the pixel at the position of the attentionpixel (the adjusted pixel value of the adjustment target pixel) in theviewpoint image subjected to the pixel shift, with the pixel value ofthe attention pixel, as in step S36 in FIG. 11.

In other words, the light collection processing unit 51 integrates thepixel value (the pixel value adjusted with the adjustment coefficientfor the attention viewpoint vp #i) of the pixel at a distance equivalentto the vector (for example, the focus shift amount DP #i×−1 in thiscase) corresponding to the focus shift amount DP #i from the position ofthe attention pixel among the pixels of the viewpoint image of theattention viewpoint vp #i, with the pixel value of the attention pixel.

The process then moves from step S78 on to step S79. After that, insteps S79 and 380, processes similar to the respective processes insteps S37 and S38 in FIG. 11 are performed.

Note that, although the reference viewpoint is adopted as the viewpointof the processing result image, a point other than the referenceviewpoint, or any appropriate point or the like within the syntheticaperture of the virtual lens, for example, can be adopted as theviewpoint of the processing result image.

<Description of a Computer to Which the Present. Technology is Applied>

Next, the above described series of processes by the image processingdevice 12 can be performed with hardware, and can also be performed withsoftware. In a case where the series of processes are performed withsoftware, the program that forms the software is installed into ageneral-purpose computer or the like.

FIG. 20 is a block diagram showing an example configuration of anembodiment of a computer into which the program for performing the abovedescribed series of processes is installed.

The program can be recorded beforehand in a hard disk 105 or a ROM 103provided as a recording medium in the computer.

Alternatively, the program can be stored (recorded) in a removablerecording medium 111. Such a removable recording medium 111 can beprovided as so-called packaged software. Here, the removable recordingmedium. 111 may be a flexible disk, a compact disc read only memory(CD-ROM), a magneto-optical (MO) disk, a digital versatile disc (DVD), amagnetic disk, a semiconductor memory, or the like, for example.

Note that the program can be installed into the computer from the abovedescribed removable recording medium 111, but can also be downloadedinto the computer via a communication network or a broadcasting networkand be installed into the internal hard disk 105. In other words, theprogram can be wirelessly transferred from a download site, for example,to the computer via an artificial satellite for digital satellitebroadcasting, or can be transferred by cable to the computer via anetwork such as a local area network (LAN) or the Internet.

The computer includes a central processing unit (CPU 102, and aninput/output interface 110 is connected to the CPU 102 via a bus 101.

When an instruction is input by a user operating an input unit 107 orthe like via the input/output interface 110, the CPU 102 executes theprogram stored in the read only memory (ROM) 103 in accordance with theinstruction. Alternatively, the CPU 102 loads the program stored in thehard disk 105 into a random access memory (RAM) 104, and executes theprogram.

By doing so, the CPU 102 performs the processes according to the abovedescribed flowcharts, or performs the processes with the above describedconfigurations illustrated in the block diagrams. The CPU 102 thenoutputs the process results from an output unit 106 or transmit theprocess results from a communication unit 108 via the input/outputinterface 110, for example, and further stores the process results intothe hard disk 105, as necessary.

Note that the input unit 107 is formed with a keyboard, a mouse, amicrophone, and the like. Meanwhile, the output unit 106 is formed witha liquid crystal display (LCD), a speaker, and the like.

In this specification, the processes to be performed by the computer inaccordance with the program are not necessarily performed inchronological order compliant with the sequences shown in theflowcharts. In other words, the processes to be performed by thecomputer in accordance with the program include processes to beperformed in parallel or independently of one another (such as parallelprocesses or object-based processes).

Also, the program may be executed by one computer (processor), or may beexecuted in a distributive manner by more than one computer. Further,the program may be transferred to a remote computer, and be executedtherein.

Further, in this specification, a system means an assembly of aplurality of components (devices, modules (parts), and the like), andnot all the components need to be provided in the same housing. In viewof this, a plurality of devices that are housed in different housingsand are connected to one another via a network form a system, and onedevice having a plurality of modules housed in one housing is also asystem.

Note that embodiments of the present technology are not limited to theabove described embodiments, and various modifications may be made tothem without departing from the scope of the present technology.

For example, the present technology can be embodied in a cloud computingconfiguration in which one function is shared among a plurality ofdevices via a network, and processing is performed by the devicescooperating with one another.

Further, the respective steps described with reference to the abovedescribed flowcharts can be carried out by one device or can be sharedamong a plurality of devices.

Furthermore, in a case where more than one process is included in onestep, the plurality of processes included in the step can be performedby one device or can be shared among a plurality of devices.

Meanwhile, the advantageous effects described in this specification aremerely examples, and the advantageous effects of the present technologyare not limited to them and may include other effects.

It should be noted that the present technology may also be embodied inthe configurations described below.

<1>

An image processing device including:

an acquisition unit that acquires images of a plurality of viewpoints;and

a light collection processing unit that performs a light collectionprocess to generate a processing result image focused at a predetermineddistance, using the images of the plurality of viewpoints,

in which the light collection processing unit performs the lightcollection process using the images of the plurality of viewpoints, theimages having pixel values adjusted with adjustment coefficients for therespective viewpoints.

<2>

The image processing device according to <1>, further including

an adjustment unit that adjusts pixel values of pixels of the images ofthe viewpoints with the adjustment coefficients corresponding to theviewpoints,

in which the light collection processing unit

performs the light collection process by setting a shift amount forshifting the pixels of the images of the plurality of viewpoints,shifting the pixels of the images of the plurality of viewpoints inaccordance with the shift amount, and integrating the pixel values, and

performs the light collection process using the pixel values of thepixels of the images of the plurality of viewpoints, the pixel valueshaving been adjusted by the adjustment unit.

<3>

The image processing device according to <1> or <2>, in which theadjustment coefficients are set for the respective viewpoints inaccordance with a distribution of transmittance that produces an effectof a predetermined lens and a diaphragm.

<4>

The image processing device according to <1> or <2>, in which theadjustment coefficients are set for the respective viewpoints inaccordance with a distribution of gain that produces a predeterminedfilter effect.

<5>

The image processing device according to any one of <1> to <4>, in whichthe images of the plurality of viewpoints include a plurality ofcaptured images captured by a plurality of cameras.

<6>

The image processing device according to <5>, in which the images of theplurality of viewpoints include the plurality of captured images and aplurality of interpolation images generated by interpolation using thecaptured images.

<7>

The image processing device according to <6>, further including:

a parallax information generation unit that generates parallaxinformation about the plurality of captured images; and

an interpolation unit that generates the plurality of interpolationimages of different viewpoints, using the captured images and theparallax information.

<8>

An image processing method including:

acquiring images of a plurality of viewpoints; and

performing a light collection process to generate a processing resultimage focused at a predetermined distance, using the images of theplurality of viewpoints,

in which the light collection process is performed using the images ofthe plurality of viewpoints, the images having pixel values adjustedwith adjustment coefficients for the respective viewpoints.

<9>

A program for causing a computer to function as:

an acquisition unit that acquires images of a plurality of viewpoints;and

a light collection processing unit that performs a light collectionprocess to generate a processing result image focused at a predetermineddistance, using the images of the plurality of viewpoints,

in which the light collection processing unit performs the lightcollection process using the images of the plurality of viewpoints, theimages having pixel values adjusted with adjustment coefficients for therespective viewpoints.

<A1>

An image processing device including:

a light collection processing unit that performs a light collectionprocess to generate a processing result image focused at a predetermineddistance, by setting a shift amount for shifting pixels of images of aplurality of viewpoints, shifting the pixels of the images of theplurality of viewpoints, and integrating the pixels,

in which, in the light collection process, the light collectionprocessing unit performs integration of pixel values of the images ofthe plurality of viewpoints, the pixel values obtained after adjustmentof the pixels of the images of the viewpoints with adjustmentcoefficients for adjusting the pixel values, the adjustment coefficientshaving been set in accordance with the viewpoints.

REFERENCE SIGNS LIST

-   11 Image capturing device-   12 Image processing device-   13 Display device-   21 ₁ to 21 ₇, and 21 ₁₁ to 21 ₁₉ Camera unit-   31 Parallax information generation unit-   32 Interpolation unit-   33 Adjustment unit-   34 Light collection processing unit-   35 Parameter setting unit-   51 Light collection processing unit-   101 Bus-   102 CPU-   103 ROM-   104 RAM-   105 Hard disk-   106 Output unit-   107 Input unit-   108 Communication unit-   109 Drive-   110 Input/output interface-   111 Removable recording medium

The invention claimed is:
 1. An image processing device comprising: an acquisition unit configured to acquire images of a plurality of viewpoints; a light collection processing unit configured to perform a light collection process to generate a processing result image focused at a predetermined distance, using the images of the plurality of viewpoints, wherein the light collection processing unit is configured to perform the light collection process using the images of the plurality of viewpoints, the images having pixel values adjusted with adjustment coefficients for the respective viewpoints, wherein the adjustment coefficients are set for the respective viewpoints to correspond to a gain distribution of transmittance, and wherein the gain distribution has a peak at a center of the plurality of viewpoints, decreases smoothly toward ends of the plurality of viewpoints, and becomes 0 between the center and each end of the plurality of viewpoints; and an adjustment unit that adjusts the pixel values of pixels of the images of the viewpoints with the adjustment coefficients corresponding to the viewpoints, wherein the light collection processing unit performs the light collection process by setting a shift amount for shifting the pixels of the images of the plurality of viewpoints, shifting the pixels of the images of the plurality of viewpoints in accordance with the shift amount, and integrating the pixel values, and performs the light collection process using the pixel values of the pixels of the images of the plurality of viewpoints, the pixel values having been adjusted by the adjustment unit.
 2. The image processing device according to claim 1, wherein the images of the plurality of viewpoints include a plurality of captured images captured by a plurality of cameras.
 3. The image processing device according to claim 2, wherein the images of the plurality of viewpoints include the plurality of captured images and a plurality of interpolation images generated by interpolation using the captured images.
 4. The image processing device according to claim 3, further comprising: a parallax information generation unit that generates parallax information about the plurality of captured images; and an interpolation unit that generates the plurality of interpolation images of different viewpoints, using the captured images and the parallax information.
 5. An image processing device comprising: an acquisition unit configured to acquire images of a plurality of viewpoints; a light collection processing unit configured to perform a light collection process to generate a processing result image focused at a predetermined distance, using the images of the plurality of viewpoints, wherein the light collection processing unit is configured to perform the light collection process using the images of the plurality of viewpoints, the images having pixel values adjusted with adjustment coefficients for the respective viewpoints, wherein the adjustment coefficients are set for the respective viewpoints to correspond to a gain distribution, and wherein the gain distribution gradually decreases from one end to the other end of the plurality of viewpoints; and an adjustment unit that adjusts the pixel values of pixels of the images of the viewpoints with the adjustment coefficients corresponding to the viewpoints, wherein the light collection processing unit performs the light collection process by setting a shift amount for shifting the pixels of the images of the plurality of viewpoints, shifting the pixels of the images of the plurality of viewpoints in accordance with the shift amount, and integrating the pixel values, and performs the light collection process using the pixel values of the pixels of the images of the plurality of viewpoints, the pixel values having been adjusted by the adjustment unit.
 6. The image processing device according to claim 5, wherein the gain distribution corresponds to at least one color component.
 7. An image processing method comprising: acquiring images of a plurality of viewpoints; and performing a light collection process to generate a processing result image focused at a predetermined distance, using the images of the plurality of viewpoints, wherein the light collection process is performed using the images of the plurality of viewpoints, the images having pixel values adjusted with adjustment coefficients for the respective viewpoints, wherein the adjustment coefficients are set for the respective viewpoints to correspond to a gain distribution of transmittance, wherein the gain distribution has a peak at a center of the plurality of viewpoints, decreases smoothly toward ends of the plurality of viewpoints, and becomes 0 between the center and each end of the plurality of viewpoints, wherein the pixel values of pixels of the images of the viewpoints are adjusted with the adjustment coefficients corresponding to the viewpoints, wherein the light collection process is performed by setting a shift amount for shifting the pixels of the images of the plurality of viewpoints, shifting the pixels of the images of the plurality of viewpoints in accordance with the shift amount, and integrating the pixel values, and wherein the light collection process is performed using the adjusted pixel values of the pixels of the images of the plurality of viewpoints.
 8. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising: acquiring images of a plurality of viewpoints; and performing a light collection process to generate a processing result image focused at a predetermined distance, using the images of the plurality of viewpoints, wherein the light collection processing unit performs the light collection process using the images of the plurality of viewpoints, the images having pixel values adjusted with adjustment coefficients for the respective viewpoints, wherein the adjustment coefficients are set for the respective viewpoints to correspond to a gain distribution of transmittance, wherein the gain distribution has a peak at a center of the plurality of viewpoints, decreases smoothly toward ends of the plurality of viewpoints, and becomes 0 between the center and each end of the plurality of viewpoints, wherein the pixel values of pixels of the images of the viewpoints are adjusted with the adjustment coefficients corresponding to the viewpoints, wherein the light collection process is performed by setting a shift amount for shifting the pixels of the images of the plurality of viewpoints, shifting the pixels of the images of the plurality of viewpoints in accordance with the shift amount, and integrating the pixel values, and wherein the light collection process is performed using the adjusted pixel values of the pixels of the images of the plurality of viewpoints.
 9. An image processing device comprising: an acquisition unit configured to acquire images of a plurality of viewpoints; a light collection processing unit configured to perform a light collection process to generate a processing result image focused at a predetermined distance, using the images of the plurality of viewpoints, wherein the light collection processing unit is configured to perform the light collection process using the images of the plurality of viewpoints, the images having pixel values adjusted with adjustment coefficients for the respective viewpoints, wherein the adjustment coefficients are set for the respective viewpoints in accordance with a gain distribution of transmittance, and wherein the gain distribution has at least two peaks at positions different from a center of the plurality of viewpoints, and changes smoothly toward ends of the plurality of viewpoints; and an adjustment unit that adjusts the pixel values of pixels of the images of the viewpoints with the adjustment coefficients corresponding to the viewpoints, wherein the light collection processing unit performs the light collection process by setting a shift amount for shifting the pixels of the images of the plurality of viewpoints, shifting the pixels of the images of the plurality of viewpoints in accordance with the shift amount, and integrating the pixel values, and performs the light collection process using the pixel values of the pixels of the images of the plurality of viewpoints, the pixel values having been adjusted by the adjustment unit.
 10. The image processing device according to claim 9, wherein the gain distribution becomes 0 between the center and each end of the plurality of viewpoints. 